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Abstract. The Symplectic Pontryagin method was introduced in a 
previous paper. This work shows that this method is applicable under 
less restrictive assumptions. Existence of solutions to the Symplectic 
Pontryagin scheme are shown to exist without the previous assumption 
on a bounded gradient of the discrete dual variable. The convergence 
proof uses the representation of solutions to a Hamilton- Jacobi-Bellman 
equation as the value function of an associated variation problem. 
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1. Introduction 

The Symplectic Pontryagin method was introduced in [7] , as a numerical 
method designed for approximation of the value function associated with 
optimal control problems. Under general conditions this value function, 
u : R d x [0, T] — > M, is a viscosity solution of an associated Hamilton-Jacobi 
equation, 

u t + H(x,u x ) = 0, mR d x(0,T), 
u{x,T) = g(x), 

where ut denotes the derivative with respect to the "time" variable t, and 
u x the gradient with respect to the state variable x. The function H : 
R d x R d -> R is denoted the Hamiltonian. This function is concave in its 
second argument when the Hamilton-Jacobi equation originates in optimal 
control. We shall therefore only deal with such concave Hamiltonians in this 
paper. The main ingredient in the Symplectic Pontryagin method is the 
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observation that, under favorable conditions, optimal paths to the control 
problem solve a Hamiltonian system associated with the Hamilton-Jacobi 
equation (11. ip . More accurately, if x : [s,T] — ► M. d is an optimal path for 
the optimal control problem with initial position (y, s), then there exists a 
dual function A : [a, T] — > M. d , such that x and A solve 

x'(t) = H x (x(t),X(t)), for s<t<T, 
x(s) = y, 

-A'(t) = H x (x(t),X(t)), for s<t<T, v ' ; 
A(T)= 9 '(*(T)). 

This is the Pontryagin principle, a necessary condition for optimality, in the 
case where the Hamiltonian function, H, is differentiable. The Hamiltonian 
is, however, often nondifferentiable, even in situations where the control and 
cost functions in the control problem are differentable, see [7] . Even though 
the Pontryagin principle may be formulated for other control problems than 
those having differentiable Hamiltonians, the version in fjl .21) is appealing as 
a starting point for numerical methods. 

In the Symplectic Pontryagin method, the Hamiltonian system (jl.2p is 
used with a regularized Hamiltonian, H 5 , satisfying \H s (x, X) — H(x, A)| < <5, 
for all (x, A) € M. 2d . The Hamiltonian system (11. 2j) makes sense when H 
has been replaced with the differentiable H s . The comparison principle 
for Hamilton-Jacobi equations then grants that the maximum difference 
between the solution to the original Hamilton-Jacobi equation (11. ip and the 
one where H s has taken the place of H is of the order 5. 

The second ingredient in the Symplectic Pontryagin method is the ap- 
plication of the Symplectic Euler numerical scheme for the Hamiltonian 
system (11. 2\\ with the regularized Hamiltonian H s . We shall for simplicity 
assume that an approximation of u(x s ,0) is to be computed. The method 
to approximate u for a starting position whose time coordinate differs from 
zero, is similar. The time interval [0, T] is split into N intervals of length 
At = T/N . We introduce the notation t n = nAt, for n = 0, . . . , N. The 
Symplectic Pontryagin scheme is: 

x n +i = x n + AtH 5 x (x n , A n+ i), n = 0, . . . , N - 1 

xo = x s , 

a (1-3) 
A n = A n+ i + AtH d (x n , A n+1 ), n = 0, . . . ,N - 1 

A^v = g'(x N ). 

In [7] this method is analyzed by extending the solutions {x n } to piecewise 
linear functions. These functions are used to define an approximate value 
function which is shown to solve a Hamilton-Jacobi equation equal to the 
initial equation, but with an additional error term. The above mentioned 
comparison principle gives the difference between the approximate and the 
exact value functions. 
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In this paper the Symplectic Pontryagin method is analyzed in a differ- 
ent way. The first step is here to consider the Hamilton-Jacobi equations 
with the original Hamiltonian H, and the regularized variant H s . As men- 
tioned above, this gives a difference of the order 5 between the corresponding 
solutions. Next, a representation formula of the solutions to the Hamilton- 
Jacobi equations as minima of a variation problem is used. It is shown that 
there exists one solution to the Symplectic Pontryagin method which is also 
a minimizer to an Euler discretized version of the variation problem. The 
minimizer to the dicretized variation problem is then shown to give a value 
which is close to the value of the original variation problem. 

This work extends the result in [7] in the following ways: 

• A solution to the Symplectic Pontryagin method is shown to exist 
for the considered class of Hamiltonians. 

• The result in [7] relies on the assumption that the variation <9A n +i/ 'dx, 
is bounded everywhere but on a codimension one hypersurface. This 
assumption is not needed with the present approach. 

• The error bound in the present paper is shown to be of the order 
5 + At, compared with the previous 5 + At + At 2 /5. The new result 
therefore ensures the possibility of decreasing 5 independently of At, 
without deteriorating the error estimate. 

The paper is organized as follows. In section[2]we present the main results 
of this paper, existence of solutions to, and convergence of, the Symplecitic 
Pontryagin method. In sections [3] and H] the convergence proof is given in 
the form of a series of lemmas. 

2. Main Results 

Given the Hamiltonian function H : R rf x M. d ->Rwe define the running 
cost function 

L(x,a) = sup { -a- X + H(x,X)}, (2.1) 

for all x and a in W d . This function is convex in its second argument, and 
extended valued, i.e. its values belong to RU {+oo}. If the Hamiltonian is 
real-valued and concave in its second variable it is possible to get it back 
when having possession of L: 

H(x,X)= inf {\-a + L(x,a)\. (2.2) 

This is a consequence of the bijectivity of the Legendre-Fenchel transform, 
see [6]. 

For Hamilton-Jacobi equations with initial data given (Cauchy problems, 
i.e. not as here with data given at time T) and convex Hamiltonians, the 
Hamiltonian and the running cost are connected via the usual Legendre- 
Fenchel transform. The results presented here could have been presented 
for Hamilton-Jacobi equations in this form, but since it is more common for 
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optimal control problems to include cost functions of the terminal positions 
than the initial positions, the current setting is chosen. 

The cornerstone in the convergence analysis for the Symplectic Pontryagin 
method will be the following representation theorem, taken from [4], see 
also [2,5,6,8]. The result is given in greater generality in that paper, but we 
present a form suited for our purposes. We will assume that the Hamiltonian 
satisfies the following estimates: 

\H(x,\i) -H(x,X 2 )\ <Ci|Ai-A 2 |, for all x,Ai,A 2 G R d , 

\H(x 1 ,X)-H(x 2 ,X)\<C 2 \xi-x 2 \(l + \X\), for all xi,x 2 , A G R d . ^ 

Then the following representation result holds: 

Theorem 2.1. Let the Hamiltonian H : R d x R d -> R be concave in its 
second argument, and satisfy the bounds in (|2.3p . Let the running cost L 
be defined by ([2,ip . and let g : M. d —>■ R be a continuous function such that 
g(x) > —k(l + \x\) for all xEff 1 for some k > 0. Then the value function 



Furthermore, if a function is a continuous viscosity solution to (11.11) which 
satisfies (12. 5j) . then this function must be the value function V . 

We also have the following result, stating that optimal solutions to the 
value function V solve a Hamiltonian system. For a proof, see [3]. 

Theorem 2.2. Let the conditions in Theorem \2.1\ be satisfied, and let (y,s) 
be any point in M. d x [0, T]. Then a minimizer x : [s, T] — > R d for V(y, s) in 
(|2.4p exists. If furthermore the Hamiltonian H and the terminal cost g are 
continuously differentiable, there exists a dual function A : [s,T] — ► R rf ; such 
that (x, A) solve the Hamiltonian system (|1.2p 

In view of Theorem 12.11 a possible approximate value of u(x s ,t m ) is the 
minimizer of 




(2.4) 



is a continuous viscosity solution to (jl.ip which satisfies 

u(x,t) > -k(l + |x|) for all (x,t) G M. d x [0,T]. 



(2.5) 



iV-l 




(2.6) 



n=m 



where a n G R d and 



x ri+ \ = x n + Ata, 



for m < n < N — 1 



(2.7) 



We define a discrete value function as the optimal value of J: 



u(x s ,t m ) = inf {J( a . 3itm) (a m; . . . ,ajv-i)|| %, • • • ,c*n-i G R d } 
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For the proof of Theorem 12.51 the existence theorem, we need two defini- 
tions. 

Definition 2.3. A function f : R d — » M. is locally semiconcave if for every 
compact convex set V C R d there exists a constant K > such that f(x) — 
K\x\ 2 is concave on V. 

There exist alternative definitions of semiconcavity, see [2] , but this is the 
one used in this paper. 

Definition 2.4. An element p € M. d belongs to the superdifferential of the 
function f : M. d — > R at x, denoted D + f(x), if 

,. f(y)- f(x)-p-(y-x) . n 

hmsup : : < 0. 

y^x \y — x\ 

Theorem 2.5. Let x s be any element in M. d , and g : R rf ->ia continuously 
differentiable function such that \g'(x)\ < L g , for all x G W^ 1 , and some 
constant L g > 0. Let the Hamiltonian H : M" X M. d — > R be two times 
continuously differentiable and concave in its second argument, and satisfy 
the bounds in (|2.3p . Let the running cost L be defined by (|2.f p . Then 
there exists a minimizer (a TO , . . . , ajv-i) of the function J( Xs ,t m ) ^ n (|2.6p . 
Let (x m , . . . , xn) be a corresponding solution to (]2.7p . Then there exists a 
discrete dual variable X n , n = m, . . . , N satisfying 

x n+ i = x n + AtH x (X n +i,x n ), for all m < n < N - 1, 

%m — X s 

(2 8) 

\ n = \ n+ i + AtH x (X n+ i,x n ), for all m < n < N - 1, 
Aat = g'{x N ). 
Hence a n = H\(\ n +i,X n ) for all m < n < N — 1. 

Proof. Step 1. By the properties of the Hamiltonian it follows that the 
running cost L is lower semicontinuous, see [6]. It is clear that the infimum 
of J(x B ,tm) 1S l ess than infinity. Let e > and o4, n = m, ... ,N — 1 be 
discrete controls such that 

J(x s ,t m )( a mi • • • i a N-l) 

< inf { J (a , s tm) (a m , . . . ,ajv-i) | «n G R d , n = m, . . . ,N - 1} + e. 

Since the Hamiltonian H satisfies the bounds (|2.3[) , it follows that all a e n are 
bounded by C\. Hence there exist {£»} and corresponding {a^} such that 
£j — > 0, and — > a n , where \a n \ < C\, for all n. The lower semicontinuity 
of L implies that this {a n }^I^j is a minimizer for J( x . Sj o)- 

Step 2. Assume that u(-,t n+ i) is locally semiconcave, and that A n +i G 
D + u(x n +i, t n+ i). We will show that this implies 



A n +i • «n + L(x n ,a n ) = H(\ n+ i,x n ). 



(2.9) 
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By the semiconcavity we have that there exists a constant A > 0, such 
that 

u(x n + Ata, t n+ i) < u(x n+1 ,t n+1 ) + AtA n+ i • (a - a n ) + A\a - a n \ 2 . (2.10) 

for all a in a neighborhood around a n . Since we know that the function 

a i— > u(x n + Ata, t n+ i) + AtL(x n , a) 

is minimized for a = a n , the semiconcavity of u in (|2.10[) implies that the 
function 

a i — > X n +i ■ ol + A\a — a n \ 2 + L(x n , a) (2-H) 
is also minimized for a = a n . We will prove that the function 

a h-> A n+ i • a + L(x n , a) ( 2 -12) 

is minimized for a = a n . Let us assume that this is false, so that there 
exists an a* G and an e > such that 

A„.+i • a n + L(x n , a n ) - X n+1 ■ a* - L(x n ,a*) > e. (2.13) 

Let ^ G [0, 1], and a = ^a* + (1 — £,)a n . Insert a into the function in (|2.1ip : 

A n+ i • a + A\a — a n \ 2 + L(x n , a) 

= £A n+ i • a* + (1 - OVfi • «n + Af\a* - a n \ 2 + L(x n , £a* + (1 - £)a n ) 
< ^A n+ i • a* + (1 - C)A n +l • OLn + M 2 1 a* - «n| 2 + a*) + (1 - O-^C^n, a n ) 

< A„ + i • a n + L(x n , a n ) + A^ 2 |a* - a n \ 2 - £e 

< A„+i ■ a n + L( 

for some small positive number £. This contradicts the fact that a n is a 
minimizer to the function in (|2.1ip . Hence we have shown that the function 
in (I2.12|) is minimized at a n . By the relation between L and H in (12. 2D our 
claim (12. 9h follows. 

S"iep 5. From the result in Step 2, equation (12. 9p . and the definition of the 
running cost L in (|2.ip it follows that a n = H\(x n , A n+ i), for if this equation 
did not hold, then A n+ i could not be the maximizer of — a n ■ A + H(x n , A). 

Step 4- Ln this step, we show that if u(-,t n+ i) is locally semiconcave, 
and A n+ i G D + u{x n+ \, i n +i)> then u{-,t n ) is locally semiconcave, and A n G 
D + u(x n ,t n ). 

By the assumed local semiconcavity of u{-, t n+ \) we have that there exists 
a K > such that 

u(x,t n+ i) < A n+ i • (x - x n+ i) + K\x - x n+ i| 2 , 

for all x in some neighborhood around x n+ \. Let us now consider the control 
H\(x, A ra+ i) at the point (x, t n ). Since this control is not necessarily optimal 
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except at (x n ,t n ), we have 

u(x,t n ) < u[x + AtH x (x, X n+ i),t n+ i) + AtL(x,Hx(x,X n+ i)) 
< u(x n+ i,t n+ i) + A n+ i • (x + AtH\(x, A n+ i) - x n+1 ) 
+ K\x + AtH x {x, A n+ i) - x n+1 \ 2 + AtL(x, H x (x, X n+1 )) , (2.14) 

for all x in some neighborhood around x n . By the definition of L in (|2.ip it 
follows that 

L(x, #a(z, A„+i)) = -H x (x, X n +i) • A n+ i + i/(x, A n+ i). 
With this fact in (|2.14p we have 

u(x, t n ) < u(x n +i,t n +i) + A n+ i • (x - x n+ i) 

+ K\x + AtH x (x,X n+1 ) -x n+1 \ 2 + AtH(x,X n+1 ). (2.15) 
By the result in step 3 we have that 

x n +i = x n + AtH x (x n , A n+ l), 
so that, by the fact that H is twice continuously differentiable 

\x + AtH x (x, A n+ i) - x n+ i| 

= |x - x n + At(H x (x, X n+ i) - H x (x n , Xn+i)) | < if |a; -x n \, (2.16) 

for some (new) K and x in a neighborhood of x n . We also need the facts 
that 

u(x n , tn) = u(x n+1 ,t n+1 ) + AtL(x n , a n ) (2.17) 

and 

L(x n , a n ) = -X n +i ■ & n + H(x n , X n +i) = -X n +i ■ + H(x n , X n +i). 

(2.18) 

We insert the results (I27T61) . (I2TT71) . and OTT8]) into (I2TT51) . This implies that 
there are constants K > and K' > 0, such that 

n(x,t n ) 

< u(x n , t„) + A n+ i • (x - x n ) + At(H(x, X n +i) - H(x n , X n+ i)) + if | x x n i 

< n(x n , tn) + (A n+ i + AtH x (x n , X n +i)) ■ (x - x n ) + K\ x x n i 

— w(x n , £ n ) -(- A n • (x x n ) K\x x n \ , 

for all x in a neighborhood around x n . We here used the differentiability of 
H in the last inequality, and the A ra evolution in (|2.8p in the last equality. 
This shows that u(-,t n ) is locally semiconcave, and that A ra £ D + u(x n ,t n ). 

Step 5. Since u(x,T) = g(x) and g is differentiable, it follows that u(-,T) 
is semiconcave. Induction backwards in n shows that u(-, t n ) is locally semi- 
concave for all n. Thereby the conclusions in step 2 and 3 hold for all n. 

□ 
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By the results from sections [3] and IH Lemmas 13.71 and 14.21 we have the 
following convergence result. 

Theorem 2.6. Let the Hamiltonian H be concave in its second argument, be 
continuously differentiable and satisfy f|2.3|) . Let g : M. d — > R be differentiable 
and satisfy \g'(x)\ < C3, for all x 6 M d . Let x s be any element in M. d . Then 
the viscosity solution u satisfying the conditions in Theorem \2.1l and the 
approximate value function u satisfy 

_ ChChT^ eC2T _ 1)M + ^ _ ChCz {eC2T _ 1)At 

< u(x s ,0) - u(x s ,0) 

< \c x C 2 {C z + l)e C2T TAt. 

If the Hamiltonian is not differentiable, the following result may be used. 

Theorem 2.7. Let H be a Hamiltonian which is concave in its second ar- 
gument, and satisfies (|2.3p . Let H s be a Hamiltonian that satisfies the same 
conditions, and which is continuously differentiable, and satisfies 

\\H - H s \\ L oo( R d xR d} < 5. 

Then 

_ ChC^T ((gC72T _ i)At + _ C^ {eC2T _ 1)M _ T6 

< u(x s ,0) - u(x s ,0) 



< ^CiC 2 (C 3 + l)e C2T TAt + T6, 



where u is computed using H & 



Proof. The comparison principle for solutions to Hamilton-Jacobi equations 
gives that the two solutions to (jl.ip with H and H s differ by T5, see [7]. 
The result follows from Theorem 12.61 □ 



3. Lower bound of u(x s ,0) -u(x s ,0) 

In order to be able to prove the inequality, we need the result in Theorem 
for convex functions. The theorem uses the following definition. 



Definition 3.1. Let f be a convex real-valued function defined on M. d . A 
vector y 7^ is called a direction of recession of f if f{x + ty) is a nonde- 
creasing function of t, for every x € M. d . 

It is shown in [6] that if f{x + ty) is a nondecreasing function of t for one 
x € W 1 , then this property holds for all x S The following thorem is 
taken from [6], but in a slightly simplified form. 
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Theorem 3.2. Let |/j | i 6 /}, where I is an arbitrary index set, be 
a collection of real-valued convex functions on M. d which have no common 
direction of recession. Then one and only one of the following alternatives 
holds: 

a) There exists a vector x <EM. d such that 

fi(x) < 0, for all i € /. 

b) There exist non-negative real numbers li, only finitely many non-zero, 
such that, for some e > 0, one has 

kfi{x) > e, for all x £ R d . (3.1) 

Lemma 3.3. Assume that the Hamiltonian H satisfies equation (|2.3p and 
is concave in its second argument. Let x\, x 2 , and a\ be any elements in 
R d , and (3 any positive number. Then there exists an element a 2 in M. d such 
that 

|«2 - cti\ < C 2 \x 2 - xi\ + (3, (3.2) 

and 

L(x 2 ,a 2 ) < L(x\,ai) + C 2 \x 2 - x\\. (3.3) 

Proof. If L{x\,a\) = oo, we can let a 2 = a±. We henceforth assume that 
L(xi,ai) < cxd. From the definition of L in (|2.ip it follows that H(x\,0) < 
L(x\,ai). By (|2.3p we have 

H(x 2 ,0) < L(xi,a 1 ) + C 2 \x 2 -x 1 \. (3.4) 

We start by assuming that a strict inequality holds in (|3.4p . and then use 
the result for this case to prove the result in the general situation where we 
also allow equality in the equation. 

When we assume that H(x 2 ,0) > L(xi,a>\) + C 2 \x 2 — x\\, it is possible 
to use Theorem 13.21 with the functions 

A i— > —H(x 2 , A) + a ■ A + L(x\,ai) + C 2 \x 2 — x\\. (3-5) 

We let a be any element in the ball of radius C 2 \x 2 — x\\ + (3 centered at 
a\, the set we will use as the index set I in Theorem 13.21 

We now show that the functions in (|3.5p have no common direction of 
recession. Let A = av, where v is a unit vector in M. d , and let a = a\ + 
(C 2 \x 2 — x\ \ + f3)v . The corresponding function from (|3.5p then satisfies 

- H(x 2 , av) + aa\ ■ v + [C 2 \x 2 - x\\ + j3)a + L(x\, a\) + C 2 \x 2 - x\\ 
= —H(xi,av) + act\ ■ v + [C 2 \x 2 — x\\ + (3)a + H{x\,av) — H(x 2 , av) 
+ L(x±, a\) + C 2 \x 2 — x\\ > (3a — > oo when a — > oo, 

(3.6) 

since L(x\, ol\) > — aa± ■ v + H{x\, av) by the definition of L in (12. ip . and 
H(xi,a • v) — H(x 2 ,a ■ v) > —C 2 \x 2 — xi\(l + a) by equation f)2.3|) . 



10 



MATTIAS SANDBERG 



We can therefore apply Theorem 13.21 on the set of functions in (|3.5p with 
the a:s taken from the ball of radius C 2 \x 2 — x\ | +/?, centered at ot\ . Since all 
functions in f|3 . 5f) are positive at A = 0, and at least one function is positive 
at every other point, by (|3.6f) . alternative a) in Theorem 13.21 can not hold. 
Therefore the set of functions satisfy alternative b). We may assume that 
the numbers U in f)3. If) satisfy 

This corresponds to a multiplication of the inequality (|3.ip by a positive 
number. This changes the number e, but that is not important here. Hence 
we have that there exist a finite number of vectors a 1 , with 

\a % — a\\ < C 2 \x 2 — x\\ + /3, (3-7) 

such that 

^2h{- H(x 2 ,X) + a 1 ■ A + L(xi,ai) + C 2 \x 2 - xi\) 

i 

= -H(x 2 ,X) + ( hoi 1 ) ■ A + L(x 1 ,ai) + C 2 \x 2 - xi\ > e. 

i 

Since every vector a 1 satisfy (|3.7h . so does the convex combination 

i 

This convex combination can be taken as the a 2 in the lemma. 

Let us now consider the remaining case where H(x 2 ,0) = L(xi,a\) + 
C 2 \x 2 — x\\. We now modify the functions in (|3.5I) to be 

A 1 ^ -H(x 2 ,X) + a ■ A + L(xi,«i) + C 2 \x 2 — xi\ +7, (3.8) 

where 7 is a positive number. The same analysis as for (|3.5[) shows that there 
is a vector a 1 whose distance from a\ is less than or equal to C\x 2 — x\\ + (3, 
such that 

-H(x 2 ,X) + a 1 ■ A + L(xi,ai) + C 2 \x 2 - xi\ +7 > 0. 

Since all a 7 are contained in a compact set it is possible to find a sequence 
{7n}i° converging to zero such that a 7n converges to an element we may 
call a 2 . It is straightforward to check that this a 2 satisfies the conditions in 
the lemma. □ 

The idea is now to use Lemma 13.31 in order to show that there on each 
interval (t n ,t n +i) exists a function a(t), which satisfies 

\a(t)-a(t)\ < C 2 \x n -x(t)\+(3, 
L(x n ,a(t)) < L(x(t),a(t)) + C 2 \ 

(*)l, 

where the function x(t) is a minimizer to the value function V in (|2.4p . While 
Lemma 13.31 provides the existence of such an a(t) for each particular time 
t, it does not provide us with a measurable function a. In order to show 
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that such a measurable function exists, the result in Theorem 13.51 Michael's 
Selection Theorem, is needed. It can be found in a more general form in 
e.g. [1]. Before stating it we need the following definition, which can also be 
found in a more general form in [1]. 

Definition 3.4. A function F from the interval (ti,t 2 ) into the nonempty 
subsets of M. d is lower semicontinuous at t* G (ii, ^2) if f° r any z * £ F(t*) 
and any neighborhood N(z*) of z* , there exists a neighborhood N(t*) oft* 
such that for all t G N(t*), 

F(t)r\N(z*) ^ 0. 

We say that F is lower semicontinuous if it is lower semicontinuous at every 

t G (h,t 2 ). 

Theorem 3.5. Let the set-valued function F from the interval (ti,t 2 ) into 
the closed convex subsets ofM. d be lower semicontinuous. Then there exists 
f ■ (h,t 2 ) -» M d which is a continuous selection of F, i.e. which satisfies 
f(t) G F{t) for all t 1 <t<t 2 - 

Lemma 3.6. Let (3 be any positive number, x n any element in the 
terminal cost g and the Hamiltonian H be continuously differentiate, and H 
be concave in its second argument and satisfy (|2.3|) . Let x : [0, T] — > M. d be a 
minimizer for the value V(x s , 0), defined in (|2.4p . Let A(t) fort n < t < t n+ \ 
be the set of all a G M. d , such that the following conditions are satisfied: 

\a-x'(t)\<C 2 \x n -x(t)\+P, (3.9) 

and 

H(x n ,X) <a - X + L(x(t),x'(t)) + C 2 \x n - x(t)\, forall\eR d . (3.10) 
The set-valued function A(t) is lower semicontinuous. 

Proof. We use the notation a(t) = x'(t). By Theorem 12.21 we know that x 
solves the Hamiltonian system (II. 2j) . Hence a is continuous. Let t n < t* < 
t n+ \ and a* be an element in A(t*). Similarly as in the proof of Lemma l3.31 
we have that 

H(x n ,0) <L(x(t*),a(t*)) + C 2 \x n - x(t*)\. (3.11) 

We first consider the case where there is equality in (|3,lip . and later the 
case of strict inequality. 

Case 1. We assume here that 

H(x n ,0)=L(x(t*),a(t*))+C 2 \x n -x(t*)\, (3.12) 

and let t n < t < t n+ \. We know by Lemma 13.31 that A(t) is nonempty. In 
order for equation A3. 10f> to be satisfied for an element in A(t), we must have 

L(x(t),a(t)) +C 2 \x n -x(t)\ > H(x n ,0) = L(x(t*),a(t*)) + C 2 \x n - x(t*)\. 

(3.13) 

We now show that there exists an element in A(t) which is close to a* 
when \t — 1*\ is small. First note that since a* satisfies (I3.10P at t*, and we 
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Figure 3.1. The circle has radius C 2 \x n — x(t)\ + (3. When 
the dimension d is two, the set A(t) therefore consists of 
the points within the circle that satisfies (I3.10p . Since the 
set of points that satisfy this equation is convex, and a* 
is one of them, it follows that if the point a in the figure 
belongs to A(t), then so does b. We therefore see that there 
exists a point in A(t) that is no farther away from a* than 
c, i.e. a distance y/\a* — a(t)\ 2 — Cffixn ~ x (t)\ + fl) 2 - The 
situation in higher dimension is analogous. 

have assumption (|3.12p . it follows that a* = H\(x n ,0). By equation (|3.13p 
it follows that a* satisfies (|3.10p also at t. The set of a:s that solve (|3.10p 
with t fixed is convex. This convexity together with the fact that a* satisfies 
(|3.10p . and the existence of an element in A(t), by Lemma 13.31 implies that 
there exists an element in A(t) which is not farther away from a* than 

y]\a*-a{t)\>-Ct(\x n -x(t)\+(3y, (3.14) 
see Figure 13.11 We now use that 

\a(t) - a*\ < \a(t) - a(t*)\ + \a(t*) - a*\ < C 2 \x n - x(t*)\ + (3 + o(\t - t*\) 
and 

\x n ~ x(t)\ > \x n - x(t*)\ - \x(t*) - x(t)\ > \x n - x(t*)\ - 0(\t - t*\) 

in (13.141) to see that we can make the distance from a* to an element in A(t) 
arbitrarily small if \t — 1*\ is small. The case where the dimension d = 1 can 
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not be treated as in Figure 13.11 but then the same line of reasoning shows 
that there is an element in A(t) which is a distance 

max{|a(t) — a* \ — C\x n — x(t)\, 0} 

from a*. It follws that A is lower semicontinuous at t*. 

Case 2. We assume here that H(x n , 0) < L(x(t*), a(t*)) + C 2 \x n - x(t*)\. 
As in the proof of Lemma l3.3l there exists an a& which satisfies \a& —a(t*)\ < 
C 2 \x n — x{t*) \ + (3, and an e > such that 

a* ■ X + L(x(f ),a(t*)) + C 2 \x n - x{t*)\ < H(x n ,\)+e, for all A e R d . 
The linear combination + (1 — where < £ < 1, satisfies 

(ta* + (1 - i)a*) ■ A + L(x(t*),a(t*)) + C 2 \x n - x(t*)\ < H(x n , A) + fe, 

for all A £ M d . Along the optimal path the running cost satisfies 

L(x(t),a(t)) =H(x(t),X(t)) -X(t)-H x (x(t),X(t)). 

This follows by the fact that a(t) = H\(x(t), A(t)) . Since the Hamiltonian 
is assumed to be continuously differentiable, we therefore have 

L(x(t),a(t)) +C\x n -x(t)\ - (L(x(t*),a(t*)) +C\x n -x(t*)\^ \ < o(\t-t*\). 

If £ is small enough, a = + (1 — £)a* therefore satisfies 

a ■ A + L(x{t),a(t)) + C 2 \x n - x(t)\ < H(x n ,X), for all XeR d 

Since a satisfies equation (13. lQf) . similar reasoning as in case 1 shows that 
the distance from a* to an element in A{t) can be made arbitrarily small 
when \t — t*\ is made small. □ 

We are now ready to state the error estimate. 

Lemma 3.7. Let the Hamiltonian H be continuously differentiable, con- 
cave in its second argument, and satisfy (|2.3j) . and the terminal cost g be 
continuously differentiable with \g'(x)\ < C3, for all x E M. d . Then 

u(x s , 0) - u(x s , 0) > _^I(( e <* r - I) At + At 2 ) - ^( e ^ - 1) At. 

(3.15) 

Proof. As in Lemma 13.61 we let x : [0, T) -> R d be a minimizer for the value 
V(x s ,0), defined in (|2.4p . and denote x'(t) by a{t). It is easy to check that 
the set-valued function A(t) in Lemma 13.61 has closed and convex values. 
Hence Theorem 13.51 and Lemma 13.61 show that for every (3 > and any 
x n G M. d , there exists a measurable function a(t), such that for t n < t < t n+ i 

< C 2 |x„ -x{t)\ +(3, 

L(x n ,a(t)) < L(x(t),a(t)) + C 2 \ (t)\. 
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iJV-1 „„j f~ iJV-1 



We now define the vectors {x n } n=0 and {a n } n= o as follows. Let xo = x s . 
Then there exists a a : (to s ii) — ► as above. Let 

ti 



l P 1 
5 ° = At J "W dt ' 



and xi = xo + Afdo- This xi now takes the role of x\ in Lemma 13.61 We 
can therefore find a : (t\,t 2 ) — > M d , and define dt\ and x 2 as before, and 
then iterate this process further. 

Step 1. This step consists of a proof that 

A 

max \x(t n ) -x n \ < (e CT - 1)(-At + S/C). (3.16) 

0<n<N v 2 ' 

To see this, we write 

rt n 

x(t n ) - x n = x(t n -i) - x n -i + / (a(t) - a{t))dt. (3.17) 



tn-l 



We now use that 



|a(i)-a(t)| < C 2 |x„_i-x(t)|+/3 < C 2 \x(t n - 1 )-x n - 1 \+C 2 \x(t)-x(t n -. 1 )\+p t 

(3.18) 

when i n _i <t<t n . We now use that \x'(t)\ < d by (p3|) . and (j37T8j) in 
(13171) to obtain 

C C 

\x(t n ) - x n \ < (1 + C 2 At)\x(t n _ l ) - x n -i\ + ~Y^^t 2 + P*t 

< (1 + C 2 At) 2 jx(t n _ 2 ) - x n _ 2 j + ((1 + C 2 At) + 1) (^At 2 + pAt) 

< ... < (l + C 2 At) n [x(0) -5 | 

+ ((1 + C 2 At) n - 1 + (1 + C 2 At) n ~ 2 + . . . + 1) (^At 2 + /3At) 



r>A/ , „ At 2 + /?At) = ((l + C 2 Atr-l)( 1 iA i + /3/C 2 ) 

< e^/^Ai + /3/C 2 ) < (e<^ - l)(^At + (3/C 2 ), 

where the second last inequality follows from At = T/N, and the second 
last equality follows from x(0) = xq, and the formula for a geometrical sum. 

Step 2. We now provide a lower bound for the term f Q L(x(t),a(t))dt, 
present in u(x s , 0). For this purpose we use that for t n < t < t n +i, 

L(x n ,a(t)) < L(x(t),a(t)) + C 2 \x n - x(t)\ 

< L(x{t),a(t)) + C 2 \x n - x(t n )\ + C 2 \x(t) - x(t n )\. (3.19) 
From the boundedness of |x'| < C\, we have 

*n+l Q 

\x(t) - x(t n )\dt < -j At 2 (3.20) 
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We now use the result from step 1, equation (|3.16j) . together with (13.190 and 

r T N-l r rt 



/ L(x(t),a(t))dt>^2( L(x n ,a(t))dt 



- C 2 T(e c > T - l)(^Ai + (3/C 2 ) - ^^At 2 
> At ^ L(x n , a n ) - C 2 T(e c * T - 1) At + (3/C 2 ) - ^^At 2 (3.21) 

n=0 

Step 3. Since 

N-l 



K x s,0) < At L(x n , a n ) + ff(5jv) 



n=0 

we can now achieve the desired lower bound, 
u(x s ,0) - u(x g ,Q) 

>AtJ2 L{xn, a n )-C 2 T{e c * T - 1) At + /3/C 2 ) - At 2 + g (x(T)) 

n=0 

JV-1 

- (At ^2 L{x n ,a n ) + g(xN)) 

n=0 

> -C 2 T(e c * T - l)(^At + (3/C 2 ) - ^^At 2 - C 3 |x(T) - x*| 
>_C a T(e* T -l)(^At4^ 

Since the number (3 can be made arbitrarily small, the lower bound (|3.15p 
follows. □ 



4. Lower bound of u(x s ,0) -u(x s ,0) 

In order to be able to prove this lower bound, we first need to establish a 
bound on the discrete dual variable. 

Lemma 4.1. Suppose g : R d — > R is differentiable and satisfies \g'(x)\ < C3 
for all x G R rf . Suppose H : M. d x R rf — > R is continuously differentiable 
and satisfies (]2.3j) . Zet {x n , A n }^ =0 6e an optimal discrete solution and dual 
which satisfy (j2.8j) . T/ien 

I A„ I < (C 3 + l)e C2T - 1. (4.1) 
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Proof. 

AtH x (x n ,X n+1 )\ < (l + C 2 At)|A n+ i| + C 2 At 
<...<(! + C 2 At) N - n \X N \ + ((1 + C 2 At) N - n - 1 + ... + l)C 2 At 

< C 3 (l + C 2 Atf + {l + C f t S~ l C 2 At 

< C 3 e C2T + e c ' 2T - 1. 

□ 

Lemma 4.2. Let the conditions in Theorem \2.1\ be satisfied. Let g : R — > R 
6e differentiate and satisfy \g'(x)\ < C3, /or every x € R d . Xei H be 
continuously differentiate. Then, for any x s € R d , 

u(x s ,0)-u(x s ,0) > ~dC 2 (C 3 + l)e C2T TAt. 

Proof. The approximate value function is given by 

N-l 

u(x s ,0) = At L(x n , a n ) + 3(scjv), 

n=0 

for some {x n ,a n }. The solution {x n }^ =0 is extended to a piecewise linear 
function x : [0, T] — ► R d , defined by 

x[t) = ^ x n+1 + — — — x n , when t n < t < t n+x . 

Using this extended approximate solution, and the fact that 

g(x N ) = u(x N ,T), 

we can represent the difference between the original and the approximate 
value functions as 

N-l 



u(x Q , 0) - u(x , 0) = At L{x n ,a n ) + u(x{T),T) - «(z(0), 0) 

n=0 

W- 1 rT d 

= At^ L(x n , an) + / -u(x(t),t)dt. (4.2) 

n=0 ^° 

The last equality in the above formula follows by the fact that the function 
1 1 — > t) is absolutely continuous since x(t) is Lipschitz continuous, and 

u is locally semiconcave. That u is locally semiconcave is shown in [2] for a 
slightly less general case than is considered here, but the proof generalizes 
easily to the present conditions. 

We now introduce the function H : R d x R d —> R, defined by 

H(x, A) = H(x, A) - k( max (|A| - ((C 3 + l)e C2T - 1), 0)^ " 
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where K is a positive constant. Define the corresponding running cost 

L(x, a) = sup ( - a • A + H(x, A)) . (4.3) 

\eR d 

Because H (x, A) < H (x, X) for all x and A, it follows that L(x, a) < L(x, a) 
for all x and a. But using Lemma f4.1l we also establish the opposite inequal- 
ity for the optimal state and control variables: 

L(x n ,a n ) > -a n -X n+ i+H(x n , X n+ i) = -a n -X n+ i+H(x n , A n+ i) = L(x n ,a n ). 

Hence we may exchange L with L in (|4.2p . 

In order to bound the integral in (|4.2j) from above, we now use the local 
semiconcavity of the value function u. This property implies that the su- 
perdifferential D + u(x(t),t} is nonempty for all < t < T. Let p = (p x ,Pt) 
be an element in D + u[x(t), tj . Let t G (0, T) satisfy t ^ t n for re = 0, . . . , N, 
so that x is differentiate at t. We split the difference quotient approximat- 
ing the backward derivative at t: 

u(x(t),t) — u[x(t — h),t — /i) 
h 

= - [u(x(t -h),t-h) - u(x(t),t) -pt(-h) -p x ■ (x(t - h) - x(t))]/h 

x(t) - x(t - h) 

+ Pt+Px- ^ " 

The semiconcavity of u implies that there is a constant K, such that the 
quotient involving the square bracket above is greater than or equal to 

-K(h 2 + (x(t) -x{t-h)f)/h. 

When h is small enough, the difference x(t) — x(t — h) = a n , for some n. If 
we temporarily let d/dt denote the backward derivative, we therefore obtain, 
when letting h — > 

iu(x(t),t)>pt+p x - an. (4.4) 
at 

The double sided and the backward derivative of u(x(t),t) differ on a set 
of measure zero, so there is no problem in using the backward derivative in 
(|4.2p . By the relation (|4.3p between the running cost and the Hamiltonian, 
the following inequality holds: 

p x ■ a n + L(x(t),a n ) > H(x(t),p x ). (4.5) 

The fact that u is a viscosity solution of (jl.ip implies that 



Pt + H(x(t),p x ) >0. 



(4.6) 
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When combining equations 114. 41) . (|4.5|) . and (14. 6|) with the error representa- 
tion (pi~2|l . we find that 

AtL(x n ,a n ) - L(x(t),a n ) dt) 

n=0 ^ tn 

I l"n-\-l _ 

= ^2 (L(x n ,a n ) - L(x(t),a n ))dt. (4.7) 

n=0 *" 

With the quadratic term added to the Hamiltonian, the supremum in the 
definition of L in (|4.3p is always attained. Denote by A* a maximum point 
in A for the coordinates (x(t),a n ), i.e. 

L(x(t),a n ) = -otn ■ A* + H(x(t), A*). 

We thereby have the lower bound 

L(x n ,a n ) - L[x(t), a n ) > -a n ■ A* + H(x n , A*) - ( - a n • A* + H(x(t), X*^j 

= H(x n , A*) - #(x(t), A*) = H(x n , A*) - H(x(t), A*). (4.8) 

It is straightforward to show that for each e > 0, there exists a X, such that 
every maximizer A* in the definition of L, (|4.3I) . satisfies 

I A* | < (C 3 + l)e c ' 2T -l + e. (4.9) 

When we combine the bound on H in (f!Oj) with (JITTj) . (|4T8j) . and (|4T9j) . we 
obtain 

u(x ,0) - u(x ,0) > -^CiC 2 (C3 + l)e C2T TAt. 

□ 
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